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Abstract 

Feature-based approaches have recently become very popular in computer vision and image analysis applications, 
and are becoming a promising direction in shape retrieval. SHREC 1 1 robust feature detection and description 
benchmark simulates the feature detection and description stages of feature-based shape retrieval algorithms. 
The benchmark tests the performance of shape feature detectors and descriptors under a wide variety of trans- 
formations. The benchmark allows evaluating how algorithms cope with certain classes of transformations and 
strength of the transformations that can be dealt with. The present paper is a report of the SHREC 11 robust 
feature detection and description benchmark results. 

Categories and Subject Descriptors (according to ACM CCS): H.3.2 [Information storage and retrieval]: Information 
Search and Retrieval — Retrieval models 1.2.10 [Artificial intelligence]: Vision and Scene Understanding — Shape 



1. Introduction 

Feature-based approaches have recently become very popu- 
lar in computer vision and image analysis applications, no- 
tably due to the works of Lowe [Low04], Sivic and Zisser- 
man [SZ03], and Mikolajczyk and Schmid [MS05]. In these 
approaches, an image is described as a collection of local 
features ("visual words") from a given vocabulary, resulting 
in a representation referred to as a bag of features. The bag 
of features paradigm relies heavily on the choice of the local 



t Organizer of the SHREC track. All organizers and participants are 
listed in alphabetical order. For any information about the bench- 
mark, contact michael.bronstein@usi.ch. Authors are listed alpha- 
betically. 



feature descriptor that is used to create the visual words. A 
common evaluation strategy of image feature detection and 
description algorithms is the stability of the detected features 
and their invariance to different transformations applied to 
an image. In shape analysis, feature-based approaches have 
been introduced more recently and are gaining popularity in 
shape retrieval applications. 

SHREC 11 invariant feature detection and description 
benchmark simulates the feature detection and descrip- 
tion stages of feature-based shape retrieval algorithms. The 
benchmark tests the performance of shape feature detectors 
and descriptors under a wide variety of different transfor- 
mations. The benchmark allows evaluating how algorithms 
cope with certain classes of transformations and what is the 
strength of the transformations that can be dealt with. 
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This report presents a long version of the paper 
[BBB*11]. 

2. Data 

The dataset used in this benchmark was from the TOSCA 
shapes [BBK08], available in the public domain. The shapes 
were represented as triangular meshes with approximately 
10,000-50,000 vertices. 

The dataset includes ones shape class (human) with simu- 
lated transformations. Compared to the SHREC 2010 bench- 
mark, there are additional transformation classes and the 
transformations themselves are more challenging. For each 
null shape, transformations were split into 1 1 classes shown 
in Figure 1 : isometry (non-rigid triangulation- and distance- 
preserving almost inelastic deformations), topology (weld- 
ing of shape vertices resulting in different triangulation), 
rasterization (simulating non-pointwise topological artifacts 
due to occlusions in 3D geometry acquisition), view (sim- 
ulating missing parts due to 3D acquisition artifacts), par- 
tial (missing parts), micro holes and big holes, global uni- 
form scaling, global ajfine transformations, additive Gaus- 
sian noise, shot noise, down-sampling (less than 20% of the 
original points). 

In each class, the transformation appeared in five dif- 
ferent versions numbered 1-5. In all shape categories ex- 
cept scale and isometry, the version number corresponded 
to the transformation strength levels: the higher the num- 
ber, the stronger the transformation (e.g., in noise transfor- 
mation, the noise variance was proportional to the strength 
number). For the isometry class, the numbers do not re- 
flect the strength of the transformation. The total num- 
ber of transformations was 55. The dataset is available at 
http://tosca.cs.technion.ac.il/book/shrec_feat.html. 

3. Evaluation methodology 

The evaluation was performed separately for feature detec- 
tion and feature description algorithms. Feature detectors 
were further divided into point and region; feature descrip- 
tors were divided into point, region, and dense. The partici- 
pants were asked to provide, for each shape Y in the dataset, 
(i) a set of detected feature points J-(Y) = {yi^ G Y}k or re- 
gions J-(Y) = {Yi C Y}i; (ii) optionally, for each detected 
point yj, a descriptor vector {f(>'A:)}|.'^|^^' ; or alternatively, 
for each detected region 1/, a descriptor vector {f(}';)}|^j^" . 

For dense descriptors, participants provided J j . The 

performance was measured by comparing features and fea- 
ture descriptors computed for transformed shapes and the 
corresponding null shapes. 



shape Y in the dataset the groundtruth dense correspondence 
to the null shape X to be given in the form of pairs of points 

Co{X,Y) = {{x'f,,yk)}fj:i, 3 feature point yj, G T{Y) is said 
to be repeatable if a geodesic ball of radius p around the cor- 
responding point x'l^ : {x]^,yk) £Co{X,Y) contains a detected 
feature point G ^{X)^ Repeatable features are 

Tp{Y) = {y,G,F(y):.F(X)nBp(x^)^0, 

(4,y*)GCo(x,y)}, 

where Sp(xj) = {x £X : dx{x,x[) < p} and dx denotes the 
geodesic distance function in X. 

Similarly, for region detectors, a region F/ G ^(Y) is re- 
peatable if the corresponding region Xj C X has overlap 
larger than p, 

J^piy) = {^l G •^(^) : \x',nxi\/\x',uxi\ > p}. 

The repeatability of a feature detector is defined as the 
percentage | J"p(y)|/|J^(K)| of features that are repeatable, 
the definition being dependent of whether a point or region 
descriptor is used. 



3.2. Feature description 



Let {filji.'^l''", {gj)y^{^'^ denote descriptors computed on 
feature points J-(X) and J-(Y), respectively. For point de- 
scriptors, we consider as the point corresponding to the 
closest point Xj G J~{X) to x'f^, where {x'k,yk) G C{){X,Y), 
such that r^j = dx{xj,x[) < p for some p. 

Descriptor quality was evaluated using the normalized L2 
distance between descriptors at corresponding points. 



,1-^(^)1 



dkj = 



In addition, an evaluation using the ROC was performed 
as follows. The corresponding feature points Xk,yj are con- 
sidered true positives if d^j < X, for some threshold T. The 
true positive rate is defined as TPR = \{dkj < x}|/|{r^y < 
p}|; the false positive rate is defined as FPR = \{dkj < 
i}\/\{rkj > p}|. By varying the threshold x, a set of pairs 
{FPR, TPR) referred to as the receiver operation character- 
istic (ROC) curve is obtained. For a fixed FPR, the higher 
the TPR, the better. 

For a dense descriptor, the quality is measured as the av- 
erage normalized L2 distance between the descriptor vectors 
in corresponding points. 



1 



\T{X) 



3.1. Feature detection 

The quality of the feature detection was measured using 
the repeatability criterion. Assuming for each transformed 



t Features without groundtruth correspondence (e.g. in regions in 
the null shape corresponding to holes in the transformed shape) are 
ignored. 
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Figure 1: Transformations of the human shape used in the tests (shown in strength 5, left to right): null, isometry, sampling, 
rasterize, holes, micro holes, shot noise, noise, partial, view, scaling, afftne. 



4. Feature detection mettiods 
4.1. Point features 

Harris 3D (Sipiran and Bustos [SB 10]). The algorithm pro- 
poses an extension for meshes of the Harris comer detec- 
tion method [HS88]. The algorithm suggests to determine 
a neighborhood (rings or adaptive) around a vertex. Next, 
this neighborhood is used to fit a quadratic patch which is 
considered as an image. After applying a gaussian smooth- 
ing, derivatives are calculated which are used to calculate 
the Harris response for each vertex. In this benchmark, three 
different configurations were used: adaptive neighborhoods 
with 5 = 0.01, 1-ring neighborhoods , and 2-ring neighbor- 
hoods. For details, see [SB 10]. 

Mesh-DoG (Zaharescu et al. [ZBVH09] ). The method 
considers the general setting of 2-D manifolds Ml embed- 
ded in M? endowed in with a scalar function / : — >■ R, 
such as colour or curvature. This represents a generaliza- 
tion of 2-D images, that can be viewed as a uniformly sam- 
pled square grid with vertices of valence 4. Operators, such 
as the gradient and the convolution are defined in this con- 
text. A scale-space representation of the scalar function / 
is build using iterative convolutions with a Guassian kernel. 
Feature detection consists of two steps. Firstly, the extrema 
of the function's Laplacian (approximated by taking the dif- 
ference between adjacent scales - Difference of Gaussian) 
are found across scales, followed by non-maximum suppres- 
sion using a 1-ring neighbourhood both spatially and across 
adjacent scales. Secondly, the detected extrema are thresh- 
olded (400 points). Mean and Gaussian curvature computed 
using [MDSB02] were the scalar functions used for current 
tests. For exact details and settings, see [ZBVH09]. 

Mesh SIFT (Smeets et al. [MFK* 10]). The Mesh SIFT 
detector detects scale space extrema as local feature loca- 
tions. First, a scale space is constructed containing smoothed 
versions of the input mesh, which are obtained by subse- 
quent convolutions of the mesh with a binomial filter. Next, 
for the detection of salient points in the scale space, the mean 



curvature H (Mesh SIFT-H) and the principal coordinates in 
curvature space KK (Mesh SIFT-KK), which are minimal 
and maximal curvature, are computed for each vertex and 
at each scale in the scale space (Hi and KKi). Note that the 
mesh is smoothed and not the function on the mesh (H or 
KK). Scale space extrema in scale spaces of differences be- 
tween subsequent scales (dH, = — Hi for Mesh SIFT-H 
and dKKi = KKi+i - KKi for Mesh SIFT-KK) are finally 
selected as local feature locations. 

Mesli-Scale DoG (Darom and Keller [DKl 1]) We follow 
the work of Zaharescu et al. [ZB VH09] that presented a Dif- 
ference of Gaussians based feature points detector for mesh 
objects. We propose to define a Gaussian filter on the mesh 
geometry, and compute a set of filtered meshes. Consecu- 
tive octaves are subtracted to compute the DoG function, and 
define the local maxima (both in location and scale) as our 
feature points at that point and scale. In order to make the 
detected features scale invariant, we suggest to set the sup- 
port for each feature point to the width of the filter at that 
scale. For details, see [DKl 1]. 

4.2. Region detectors 

Sliape MSER (Litman et al. [LBBIO]). The algorithm finds 
maximally stable components in 3D shapes, similarly to 
the popular MSER method for feature analysis in images 
[MCUP04]. The shape is represented as a component tree 
based on vertex- or edge-wise weighting function (VW and 
EW, respectively). In this benchmark, three different weights 
were used: edge weighting by inverse of commute time ker- 
nel (EW 1/CT) and inverse heat kernel (EW 1/HKS), and 
vertex weighting by heat kernel diagonal (VW HKS). For 
details, see [LBBIO]. 

5. Feature description mettiods 
5.1. Point descriptors 

Mesli-HoG (Zaharescu et al. [ZBVH09] ). For a given inter- 
est point, the descriptor is computed using a geodesic sup- 
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port region, proportional to 3% of tiie total surface area. For 
each vertex in the neighbourhood, the 3-D gradient informa- 
tion is computed using / at the detected scale. As a first step, 
a local coordinate system is chosen, in order to make the de- 
scriptor rotation invariant. Then, a histogram of gradient is 
computed, both spatially, at a coarse level, in order to main- 
tain a certain high-level spatial ordering, and using orienta- 
tions, at a finer level. Since the gradient vectors are 3 dimen- 
sional, the histograms are computed in 3D. The histograms 
are concatenated and normalized. A 96 dimensional descrip- 
tor is obtained. The gradient of the participating neighbour- 
ing vertices is computed at the scale of the detected interest 
point. For exact details and settings, see [ZBVH09]. 

Scale Invariant Spin Image (Darom and Keller [DKl 1]) 
The Spin Image local descriptor was presented by Johnson 
and Hebert [JH99], and has gained popularity due to its ro- 
bustness and simplicity. Utilizing the local scale estimated 
by the Mesh-Scale DoG detector, we propose to derive a 
Scale Invariant Spin Image mesh descriptor, where we com- 
pute the Spin Image descriptor over the local scale estimated 
at the interest point. This improves feature point matching, 
in particular when the meshes are related significant partial 
matching. For details, see [DKl 1]. 

Local Depth SIFT (Darom and Keller [DKl 1]) The SIFT 
algorithm, presented by D. Lowe [Low04] is a state-of-the- 
art approach to computing scale and rotation invariant local 
features in images. The SIFT descriptor is based on com- 
puting a local radial-angular histogram of the pixel value 
derivatives. Inspired by Lowe's seminal work, we propose to 
compute a new local feature for 3D meshes we denote Local 
Depth SIFT (LD-SIFT). Given an interest point we estimate 
its tangent plane, and compute the distance from each point 
on the surface to that plane to create a depth map, and set 
the viewport size to match the feature scale, as detected by 
the Mesh-Scale DoG detector. This makes our construction 
scale invariant. We compute the PCA of the the points sur- 
rounding the interest point, and use their dominant direction 
as the local dominant angle, and rotate the depth map to a 
canonical angle based on the dominant angle. This makes the 
LD-SIFT rotation invariant. We compute a SIFT feature de- 
scriptor on the resulting depth map to create the Local Depth 
SIFT feature descriptor. For details, see [DKl 1]. 



5.2. Dense descriptors 

Generalized HKS (Zobel et al. [ZRHar]). The Generalized 
HKS is a generalization of the HKS [SOG09] to 1-forms 
(where a 1-form can be regarded as vector field). It is de- 
rived from the heat kernel for 1-forms in a similar way as 
the HKS is derived from the heat kernel for functions. This 
yields a symmetric tensor field of second order with a time 
parameter t. For easier comparability we consider scalar ten- 
sor invariants. For details see [ZRHar] or [ZoblO]. 
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Table 1: Repeatability (in %) at p = 5 of Mesh DoG (mean) feature 
detection algorithm. Average number of detected points: 392. 
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Table 2: Repeatability (in %) at p = 5 of Mesh DoG (Gaussian) 
feature detection algorithm. Average number of detected points: 
391. 



6. Results 

6.1. Point feature detectors. 

Tables 1-9 show the repeatability of different point descrip- 
tors at fixed radius p = 5 (approximately 1% of the shape 
diameter), broken down according to transformation classes 
and strengths. Higher repeatability scores are indication of 
better performance. Figures 2-3 show the repeatability of 
point descriptors as function of geodesic distance varying 
from to 5. 

6.2. Region feature detectors. 

Tables 10-12 show the repeatability of different point de- 
scriptors at fixed overlap of 0.7, broken down according to 
transformation classes and strengths. Figure 4 shows the re- 
peatability of region feature detectors as function of overlap 
varying from to 1 . 
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Table 3 : Repeatability (in %) at p = 5 of Mesh-Scale DoG(l) fea- 
ture detection algorithm. Average number of detected points: 3616. 
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Table 6: Repeatability (in %)atp = 5 of Mesh SIFT (KK) feature 
detection algorithm. Average number of detected points: 3786. 
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Table 4: Repeatability (in %) at p = 5 of Mesh-Scale DoG (2)fea- Table 7: Repeatability (in %)atp = 5 of Harris3D (ring 1) feature 

ture detection algorithm. Average number of detected points: 1538. detection algorithm. Average number of detected points: 449. 
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Table 5: Repeatability (in %) at p = 5 of Mesh SIFT (H) feature Table 8: Repeatability (in %) at p = 5 of Harris3D (ring 2) feature 

detection algorithm. Average number of detected points: 2564. detection algorithm. Average number of detected points: 449. 
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Table 9: Repeatability (in %)atp = 5 ofHarris3D (Adaptive) fea- 
ture detection algorithm. Average number of detected points: 449. 
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Table 10: Repeatability (in %) at overlap > 0.7 of Shape USER 
(EW l/HKS) region detector algorithm. Average mimber of detected 
regions: 12.36. 
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Table 11: Repeatability (in %) at overlap > 0.7 of Shape MSER 
(EW l/CT) region detector algorithm. Average number of detected 
regions: 8.85. 



Table 12: Repeatability (in %) at overlap > 0.7 of Shape MSER 
(VW HKS) region detector algorithm. Average number of detected 
regions: 9.25. 
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Table 13: Quality of Mesh HoG feature description algorithm (av- 
erage normalized L2 distance between descriptors at corresponding 
points) on feature points detected using Mesh DoG (mean). Average 
number of points: 392. 



6.3. Point feature descriptors 

Tables 13-18 show the performance of different point fea- 
ture description algorithms, in terms of average normal- 
ized L2 distance between corresponding descriptors. Smaller 
numbers correspond to better performance. Figure 5 shows 
the ROC curves of different point feature descriptors, using 
a fixed value of p = 5. Higher values of the vertical axis at 
a fixed point on the horizontal axis are indication of better 
performance. 



6.4. Dense feature descriptors 

Table 19 shows the performance of the GHKS dense fea- 
ture description algorithm, in terms of normalized average 
L2 distance between corresponding descriptors. Some results 
could not be computed by the participants. 
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Table 14: Quality of Mesh HoG feature description algorithm (av- 
erage normalized L2 distance between descriptors at corresponding 
points) on feature points detected using Mesh DoG (Gaussian). Av- 
erage number of points: 391. 
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Table 16: Quality of Local depth SIFT feature description al- 
gorithm (average normalized L2 distance between descriptors at 
corresponding points) on feature points detected using Mesh-Scale 
DoG (2). Average number of points: 1538. 
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Table 15: Quality of Local depth SIFT feature description al- 
gorithm (average normalized L2 distance between descriptors at 
corresponding points) on feature points detected using Mesh-Scale 
DoG (1). Average number of points: 3616. 
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Table 17: Quality of Scale invariant Spin Image feature descrip- 
tion algorithm (average normalized L2 distance between descrip- 
tors at corresponding points) on feature points detected using Mesh- 
Scale DoG (1). Average number of points: 3616. 
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Table 18: Quality of Scale invariant Spin Image feature descrip- 
tion algorithm ( average normalized L2 distance between descrip- 
tors at corresponding points) on feature points delected using Mesh- 
Scale DoG (2). Average number of points: 1538. 
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Table 19: Quality of GHKS feature description algorithm (aver- 
age normalized L2 distance between descriptors at corresponding 
points). 



[LBBIO] LITMAN R., BRONSTEIN A., BRONSTEIN M.: 
Diffusion-geometric maximally stable component detection in 
deformable shapes. Arxiv preprint arXiv: 1012.3951 (2010). 3 

[Low04] Lowe D.: Distinctive image features from scale- 
invaiiant keypoints. /JCV (50, 2 (2004), 91-110. 1,4 

[MCUP04] Matas J., Chum O., Urban M., Pajdla T.: Ro- 
bust wide-baseline stereo from maximally stable extremal re- 
gions. Image and Vision Computing 22, 10 (2004), 761-767. 
3 

[MDSB02] Meyer M., Desbrun M., Schroder P., Barr 
A. H.: Discrete differential geometry operators for triangulated 
2-dimensional manifolds. In Proceedings of VisMath (2002). 3 

[MFK'IO] Maes C, Fabry T., Keustermans J., Smeets 
D., SUETENS P., Vandermeulen D.: Feature detection on 3D 
face surfaces for pose normalisation and recognition. In Proc. 
BTAS (2010). 3 

[MS05] MIKOLAJCZYK K., SCHMID C: A performance evalua- 
tion of local descriptors. Trans. RAMI (2005), 1615-1630. 1 

[SBIO] SIPIRAN I., BUSTOS B.: A robust 3D interest points de- 
tector based on Harris operator. In Proc. Eurographics Workshop 
on 3D Object Retrieval (2010), Eurographics Association, pp. 7- 
14. 3 

[SOG09] Sun J., Ovsjanikov M., Guibas L.: A concise and 
provably informative multi-scale signature based on heat dif- 
fusion. In Eurographics Symposium on Geometry Processing 
(SGP) (2009). 4 

[SZ03] SIVIC J., ZlSSERMAN A.: Video Google: A text retrieval 
approach to object matching in videos. In Proc. ICCV (2003), 
vol. 2, pp. 1470-1477. 1 

[ZBVH09] Zaharescu a., Boyer E., Varanasi K., Ho- 
RAUD R.: Surface feature detection and description with appli- 
cations to mesh matching. 3, 4 

[ZoblO] ZobelV.: Spectral Analysis of the Hodge Laplacian on 
Discrete Manifolds. Master Thesis, 2010. 4 

[ZRHar] ZOBEL V., Reininghaus J., HOTZ I.: Generalized 
heat kernel signature. Journal of WSCG, International Confer- 
ence on Computer Graphics, Visualization and Computer Vision 
(2011 to appear). 4 



References 

[BBB*11] Boyer E., Bronstein A. M., Bronstein M. M., 
BusTos B., Darom T., Horaud R., Hotz 1., Keller Y., 
Keustermans J., Kovnatsky A., Litman R., Reining- 
haus J., SiPiRAN 1., Smeets D., Suetens P., Vander- 
meulen D., Zaharescu A., Zobel V: SHREC 2011: ro- 
bust feature detection and description benchmark. In Proc. 3DOR 
(20011). 2 

[BBK08] Bronstein A. M., Bronstein M. M., Kimmel R.: 
Numerical geometry of non-rigid shapes. Springer, 2008. 2 

[DKl 1] Darom T., Keller Y.: Scale invariant features for 3d 
mesh models. 3, 4 

[HS88] Harris C., Stephens M.: A combined corner and edge 
detection. In Proc. of The Fourth Alvey Vision Conference (19H8), 
pp. 147-151. 3 

[JH99] Johnson A. E., Hebert M.: Using spin images for ef- 
ficient object recognition in cluttered 3d scenes. IEEE Trans. 
Pattern Anal. Much. Intell. 21, 5 (1999), 433^49. 4 



© The Eurographics Association 20 11 . 



Boyer et al. / SHREC 11: robust feature detection/description 



- af f ine — G — holes — — isometry \ tnicroholes - 



sampling — & — scaling — A — shotnoise — 




Mesh DoG (mean) 



Mesh DoG (Gaussian) 




Mesh-Scale DoG (1) 



IMesh-Scale DoG (2) 



^ 50- 




4.5 5 



Mesh SIFT (H) Mesh SIFT (KK) 

Figure 2: Repeatability (%)vs distance of point feature detectors broken down according to different transformation classes. 
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Figure 3: Repeatability (%) vs distance of point feature detectors broken down according to different transformation classes. 
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Figure 4: Repeatability (%) vs overlap of region feature detectors broken down according to different transformation classes. 



© The Eurographics Assoeiatiorr 201 1. 



Boyeret al. / SHREC 1 1: robust feature detection/description 




FPR 

Mesh HoG (mean) 




FPR 

Local depth SIFT (1) 





FPR 

Mesh HoG (2) 




Local depth SIFT (2) 




Scale invariant Spin Image (1) 



Scale invariant Spin Image (2) 



Figure 5: ROC curves of point feature descriptors broken down according to different transformation classes. 
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